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Evaluation of Developmental Education Programs: 

Issues^ Problems^ and Techniques 

By Robert L. Somers 



Robert L. Somers is the Director of 
Telementoring Services, National 
Center for Developmental Education 
at Appalachian State University, 
Boone, North Carolina. 

INTRODUCTION 

Developmental education and 
other learning assistance programs are 
being called upon more and more fre- 
quently to demonstrate accountability 
to a seemingly infinite number of con- 
stituencies, both iTom within and 
from outside the institutions that 
house them. Furthfemorc, these calls, 
more often than not, are perceived as 
threatening to practitioners in the 
field. Certainly the necessity, desira- 
bility, and fairness of these calls may 
be variously debateu. Their reality, 
however, cannot. 

Inasmuch as the call for program 
evaluation has become a fact of life 
for currently practicing professionals 
in developmental education, it there- 
fore behooves professionals to respond 
in a professional manner by seizing 
every opportunity to demonstrate 
how well they are doing their jobs. 
After all, the verb "evaluate" means 
to "put a value upon." Thus, the 
evaluative process entails more than 
simply counting categories of students 
or recording gain scores: evaluation 
documents the value of professional 
efforts. 

While this article introduces some 
^ basic issues and problems pertaining 
the evaluative process in general, 
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particular emphasis is placed upon the 
evaluation of learning assistance pro- 
grams. Historical perspectives, theo- 
retical constructs, suggested designs, 
and application tips are offered not as 
an exhaustive treatment leading to the 
answer, but rather as a general discus- 
sion highlighting the techniques and 
assistance needed to comfortably per- 
form a credible job of program evalu- 
ation as local circumstances dictate. 

HISTORICAL PERSPECllVES 

This section presents a brief over- 
view of the historical context and 
cultural milieu in which the 
phenomenon of evaluation developed 
and emerged as a discrete entity. 

Origins of the evaluation process 

The evolution of the modem prac- 
tice of evaluation is a nebulous history 
—an unfortunate circumstance indeed 
for the chronicler of the phenome- 
non! Indeed, about the only thing 
most authorities agree on is that 
evaluation is the natural child of close 
cousins within the family of recently 
emerging disciplines called the ''social 
sciences." 

Citing the relative infancy of social 
science research in general, Fitz- 
Gibbon and Morris point out that Sir 
Ronald Fisher's work in statistics, "an 
essential methodological step forward 
for the social sciences," was only 
completed during the 1930's (1978, 
p. 13). Similarly, Anderson and Ball 
wryly observe that, "It has been 
debated whether program evaluation 
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qualifies as a profession, or a 
discipline, or simply an activity or 
job..." (1978, p.2i9). 

In any case, the point remains that 
one of the first and most formidable 
problems that evaluation poses both 
to the historian and to the practitioner 
is its relative novelty and immaturity 
as a process. Its emphasis, focus, and 
methods have shifted kaleidoscopically 
during its brief development. 

Program evaluation in education 

During the 1920's and 1930's, the 
ascendancy of the measurement 
movement—coupled with the bur- 
geoning social science disciplines — 
presaged the notion of educational 
program evaluation as it is known to* 
day. The works of thinkers such as 
Durkheim, Freud, and Piaget were 
ushering in\anew era. Rudimentary 
forms of subdisciplines, social and 
educational psychology among them, 
were emerging. 

Scholarly opinion is unanimous in 
crediting Ralph W. Tyler with laying 
the foundation for current practice in 
program evaluation (Anderson, 1975, 
p. 143; and Pace & Friedlander in 
Hanson, 1978, p. 2). Tyler's eight-year 
longitudinal study for the Progressive 
Education Association, which he in- 
itiated during the 1930's, merits its 
status as a seminal work for diverse 
reasons. 

Departing from the traditional 
practices of measurement (counting) 
in program evaluation, Tyler insisted 
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upon clearly defining program goals 
and objectives in behavioral tenns as 
standards against which to assess pro- 
gram effectiveness. Similarly, and no 
less significantly, he also advocated 
expansion of evaluation-design 
parameters to include the affective 
domain as well as the cognitive. 
Needless to say, these contributions 
have had far-reaching implications 
whose ultimate consequences are seen 
today,not only in program evaluation, 
but also in curriculum and instruction 
(e.g., competency-basing and criteria- 
referencing) and in learning theory 
(e.g., learning- and teaching-styles 
research and application). 

The decades following the 1930's 
also have given impetus to strides in 
program evaluation. During the 
1940's, the need of the military for 
fast and cost-effective training in time 
of national emergency spurred further 
research and application in the field 
of evaluation. The psychomotor do- 
main (e.g., flight simulators for pilot 
training) became a primary benefici- 
ary. The post-Sputnik era spurred 
evaluation efforts aimed at curriculum 
reform. Critical-path analysis, PERT 
charts, and zero-based budgeting are 
all examples of evaluative processes 
triggered by weapons development 
and by the race to the moon. Nor can 
one overlook the current proliferation 
of electronic technologies, from films 
to satellite TV, that continues to drive 
evaluation efforts to assess the effec- 
tiveness of such devices in supplanting 
traditional means of instruction. 

Finally, and not inconsequentially, 
the role of legislative mandate in fuel- 
ing educational program evaluation 
cannot be overstated. The precedent 
set by Tiile I of the 1965 Elementary 
and Secondary Education Act has 
become the standard in tying annual 
appropriations to the presentation of 
evaluative reports. 

In summary, program evaluarion 
in education developed from diverse 
roots. Grounded in the social 
sciences, it also has acquired tools and 
methods adapted by those disciplines 
from the natural sciences. Various cir- 
cumstances have shaped the process 
as it is known today. Driven both by 
practical necessity and legislarive 
mandate during the past fifty years, 
program evaluation has become en- 
trenched and insritutionalized in the 
^ducational community. 
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THEORETICAL CONSTRUCTS 

This section contains a survey of 
contemporary thinking that undergirds 
the modern practice of program 
evaluation. First, various perspectives 
on the evaluarive process are 
presented through summaries of 
representative typologies of evaluative 
designs, and then two new models 
that hold promise for the 
developmental practitioner are 
discussed in greater detail. 

IVpologies of evaluation designs 

Inasmuch as evaluation sprang 
from diverse roots and is a relatively 
recent phenomenon, it should come as 
no surprise that there are virtually as 
many classification schemata as there 
are writers and practitioners. For ex- 
ample. Pace and Friedlander identify, 
albeit with considerable ambiguity, at 
least five "schools of thought" (Han- 
son, 1978, pp. M7). 

First is the educational psychology 
model identified with figures such as 
Ebel (1965) and Thorndike and Hagen 
(1969). For this group, evaluation is 
viewed as "a judgment of merit, 
sometimes based solely on measure- 
ments. . .but more frequently involv- 
ing the sythesis of various measure- 
ments, critical incidents, subjective 
impressions, and other kinds of 
evidence" (Ebel, p. 450). 

A second type is labeled the pro- 
fessional Judgment approach . Its 
rather nebulous process relies on the 
opinions of "experts" and is the type 
of evaluation derived from the for- 
malization of school and univers\y 
accreditation procedures. 

The educational decision model is 
a relatively recent design concerned 
expressly with aiding decision makers 
in making informed choices among 
available alternatives. Alkin (1969) 
and Stufflebeam (1971) are major 
figures associated with this school of 
thcdght. 

Astin and Panos (1971) are pro- 
ponents of the educational science 
model which attempts to establish 
cause-and-effect relationships among 
observable outcomes. Its methods are 
firmly grounded in the techniques of 
educational research. 

A fifth school, the educational 
change design, applies the principle of 
inclusion from group-process theory 
to assure the participation of all con- 
srituencies who might be affected by a 
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possible departure from the status 
quo. Students, faculty, administra- 
tors, and governing boards all partici- 
pate in this design along with the 
researchers and evaluators. Its goal is 
institutional improvement. 

Other writers have struggled 
similariy with evaluation typologies. 
Dressel (1976, pp. 3-4) outlines three 
approaches roughly analogous to the 
educarional science, educational 
psychology, and professional judg- 
ment mo dels . Subsequently (pp . 
15-17), he proposes four types of 
evaluations and blurs—not without 
good cause— the traditional distinc- 
tions between formative and sum- 
mative evaluations. His schema con- 
tains planning, input, process, and 
output types, all of which he main- 
tains may be formative, summative, 
or a combination of both depending 
upon the purposes and circumstances 
of the evaluative effort. 

House (1978) identifies eight 
models while Guba and Lincoln 
(1981) insist upon distilling those eight 
into only two generic types, the 
countenance evaluation and respon- 
sive evaluation models. Tuchman 
(1979, p. 10), on the other hand, 
chooses simply to classify evaluations 
as either b^mg formative, summative, 
or ex post facto. Anderson and Ball 
(1978, pp. 3-4) wisely avoid the issue 
by focusing on purposes rather than 
models. 

Chaotic as they may be, such 
heuristic typologies do serve useful 
purposes. They attempt to establish a 
common perspective on evaluation 
and to define a common language for 
sharing ideas about the process. All 
this brouhaha about classification 
aside, what is clear is an overriding 
concern for judgments about worth or 
merit and the processes used to 
achieve those judgments. 

Boylan's four-tiered program 
evaluation model 

Two recently published works 
(Somers, Boylan, and Clowes, 1986; 
and Somers, 1987, pp. 13-14 pp. 
18-19) introduce the four levels of an 
evaluative design advocated by 
Boylan for structuring a program 
evaluation. For purposes of review 
and explication, each level is sum- 
marized verb?illy and pictorially in this 
subsectioh. (See Figure 1.) 

LevJ One: Primary tocuses on 
traditional measurement (counting) 





FIGURE ONE: 
Boylan's Model At A Glance 




LEVEL 
Primary (Counting) 


FOCUS 
How many? How much? 


EXAMPLE 
300 students in 
S courses 


Secondary (Formative^ 


What Happened? 


68<7o achieved "C" 
or above 


Tertiary (Summative) 


What was accomplished? 


12^0 decrease in 
attrition 


Serendipity 


Unanticipated benefits 
not intentional in the 
program aesign 


satisfaction with 
program generates 
more alumni revenue 



practices. Data are collected to answer 
questions framed in quantitative 
terms; "How many?" and "How 
much?" are central issues. Outcomes 
at this level might be: "The University 
of XYZ enrolled 300 students in 5 
developmental courses during the 
1978-79 academic year." 

Level Two: Secondary focuses 
upon traditionally formative ques- 
tions about program effectiveness. 
"What happened to students while 
they were in the system?" is the key 
issue at this stage. An example of an 
outcome at this level might be: "Of 
300 students enrolled in developmen- 
tal courses, 207 achieved an average 
of *C' or better in all course work at- 
tempted during the 1978-79 academic 
year." 

Level Three: Tertiary concentrates 
on the longer-term or summative ef- 
fects of the program. An outcome at 
this level might be: "The 300 students 
enrolled in developmental courses 
during the past 10 years had an attri- 
tion rate 12 percent smaller than a 
comparable non-developmental con- 
trol group." 

Level Four: Serendipity concerns 
identifying any unanticipated, gratui- 
tous program benefits that might 
emerge during the course of an 
evaluation. The single most salient 
characteristic of outcomes at this level 
is that such benefits are unintentional 
in program design; that is, the pro- 
gram was not consciously designed to 
reap such rewards. Outcomes of this 
type might be: "A five-year longitudi- 
nal analysis of alumni activities in- 
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dicates that graduates who completed 
developmental courses contribute 
more to the XYZ Foundation and 
refer more potential students to the 
admissions office because of program 
satisfaction that do graduates-at- 
large." 

Clowes's stage model for program 
evaluation 

Originally published in 1984, a 
four-stage model of program evalua- 
tion designed by Clowes for remedial/ 
developmental programs also was 
recently reviewed and assessed 
(Somers et. al., 1986; and Somers, 
1987, pp. 15-17 & pp. 18-19). It, too, 
is summarized verbally and pictorially 
in this subsection. (See Figure 2.) 

Stage One: Formative concentrates 
on quantitative methods designed to 



answer questions about what happens 
to learners while they are in the pro- 
gram. The audience at this stage con- 
sists primarily of the program staff. 
Ideally, this internal audience uses 
such data to make on-going improve- 
ments in course design. Examples of 
appropriate measures at this stage in- 
clude grades, persistence in courses, 
and ratio of hours attempted to hours 
completed successfully. 

Stage Two: Summative concen- 
trates upon finding answers to ques- 
tions about what happens to develop- 
mental learners after they have exited 
the program, i.e., "How do they fare 
in the mainstream cur'''":ulum?" This 
stage, however, is both formative and 
summative— formative in that the 
developmental program curriculum is 
assessed as a whole, and summative in 
that the effects of the overall program 
are determined. Qualitative as well as 
quantitative data are prepared for ex- 
ternal audiences. Transcripted data, 
staff qualifications, and level of pro- 
gram satisfaction are examples of 
measures appropriate at this stage. 

Stage Three: Normative also is a 
genuinely formative process to the ex- 
tent that it focuses on reassessing pro- 
gram goais, not only for the develop- 
mental or remedial program, but also 
for mainstream curricuJum programs 
fed by the former. Stage two data and 
judgments are used with both internal 
audiences (e.g., all concerned staff 
and administrators) and external au- 
diences (e.g., program advisory com- 
mittees and potential employers) to 
facilitate the process of renegotiating 
and melding the goals of institutional 
segments. 



HGURE TWO: 
Oowes's Stage Model At A Glance 

Stage One: Formative 

Stage Four: Feedback for Stage Two: Formative (Curriculum) 

Institutional Renewal and Summative (Program) Evaluation 

\ J 

Stage Three: Normative ^ 
(Goal Setting) 



Stage Four: Feedback involves 
reconsidering the measures and tech- 
niques used in the first two stages in 
light of the renegotiated program 
goals formulated in stage three. Given 
these revised goals, appropriate 
measures and techniques must be con* 
sidered to evaluate them. The primary 
tl:rust at this stage is institutional 
renewal. Ideally, all institutional con- 
stituencies (audiences) are involved as 
this stage has implications for the mis- 
sion of the institution. The process 
recommended for vse at this stage is 
comparative studies of groups of 
learners who have and have not parti- 
cipated in the developmental program. 

Boylan and Clowes: A Comparative 
Critique 

Practitioners should note that the 
differences between the Boylan and 
Clowes models make them inherently 
complementary. In consideration of 
appropriate applications for each 
design, a comparative critique of each 
follows. 

The comparative simplicity of the 
Boylan design provides an exquisitely 
ready and easy formula for both the 
novice an.' the seasoned evaluator. A 
further advantage is that it may be ex- 
ecuted more expeditiously. However, 
the Boylan model lacks the continuity 
supplied by the interfaces among the 
stages of the Clowes design. Similarly, 
Boylan's design also lacks an in- 
digenous feedback loop that, in 
Qowes's model, makes provision for 
"evaluating the evaluation." Finally, 
tied tightly as it is to internal program 
goals, Boylan's model fails to 

DO 

• Define an appropriate agenda: 
— who will be the audience? 
— what is the purpose? 

— what is to be evaluated? 
— what criteria will be used? 
— are the available data both 
adequate and appropriate? 

• Include a feedback loop 

• Prepare results appropriately for 
each "audience" who receives 
them 



capitalize on opportunities for incor- 
porating potentially useful data as 
might be externally derived, e.g., by 
the professional judgment approach. 

Because of its comparative sophis- 
tication and initial independence from 
internal program goals, the Clowes 
design takes longer to ex- 
ecute—indeed, it theoretically could 
be run indefinitely. It demands 
vigorous execution, and its potential 
for the redundant could render it 
hopelessly cumbersome. Nevertheless, 
it is this writer's contention that the 
greatest strength of the Clowes design 
derives from its realistic approach. 
Compensating for the ambiguity of 
the educational process, Clowes elo- 
quently and correctly advocates mea- 
suring program effectiveness against 
evolving, consensual goals rather than 
against the a priori goals of some 
heuristic model that is assumed to be 
both shared and understood by all 
concerned parties. 

Succinctly put, the Boylan model 
is not as powerful as the Clowes 
design, nor as ambitious. One must 
bear in mind, however, that it was not 
designed to be so. 

CONCLUDING ADMONITIONS 

Previous sections have treated 
both the theoretical and applied 
asp-cts of evaluation. By way of con- 
clusion, a few practical application 
tips are offered for developi^ental 
practitioners undertaking a pn gram 
evaluation. They are deemed »iniver- 
sally useful without regard to tne par- 
ticular design or methodologies that 
one might select. 

AVOID 

• Forcing data to support 
desired conclusions 

• Restricting too narrowly the 
area of inquiry 

• Excluding key decision 
makers 
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tion. San Francisco: Jossey-Bass, 1976. 

Tnis book presents a complete analysis of all 
phases of evaluation— formulating objectives, 
determining costs, collecting data, measuring 
program and institutional success, identifying 
undesirable effects, and recommending change. 
It specifies the purposes and requirements of 
evaluating all major aspects of higher education: 
admissions, orientation, and counseling pro- 
cesses that select and indtict students; the campus 
environment that motivates learning; instruc- 
tional methods, curriculums, courses, graduate 
and undergraduate programs; the cognitive and 
affective development of students; the non- 
instructional as well as the instructional boards; 
and the impact of state coordinating and control 
boards on institutions. 
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Federal Electric Corporation. A Programmed In- 
troduction to PERT. New York: John Wiley & 
Sons, 1963. 

PERT is the acronym for Program Evaluation 
and Review Technique and is primarily directed 
toward management and information systems 
for planning and control. This book is designed 
to present the basic knowledge necessary to use 
the technique and the applications of its rules 
and regulations in becoming a project analyst. 

Grant, Donald L. (^.). Monitoring Ongoing Pro- 
grams. San Francisco: Jossey-Bass, 1978. 

A compilation of articles stressing the impor- 
tance of evaluating ongoing programs. Infor- 
mation and examples are given for different 
kinds of programs in a variety of settings. 

Hanson, Gary R. (Ed.). Evaluating Program Ef- 
fectiveness: New Directions for Student Ser- 
vices, Vol. I. San Francisco: Jossey-Bass, 1978. 
This book discusses new methods for improv- 
ing institutional programs and services for 
students. It highli^ts successful models of new 
programs in higher education, and gives 
guidelines for implementing more effective 
evaluation of these programs. 

Love, Arnold J. Developing Effective Internal 
Evaluation. San Fsancisco: Jossey-Bass, 1983. 

Explains how any organization's survival and 
growth can be enhanced with a successful self- 
evaluation program. Uses case examples to 
discuss applications of internal evaluation in 
health fields, industry, education, and correc- 
tions. 

Loveland, Edward (Ed.). Measuring the Hard-to- 
Measure. San Francisco: Jossey-Bass, 1980. 
Describes ways of overcoming difficulties in 
program evaluation stemming from such prob- 
lems as overly abstract program objectives, hn- 
precise program definitions, and unclear 
criteria for effectiveness. Illustrates approaches 
with examples of evaluation of such aspects of 
higher education as scholarships and teaching. 

Morris. Lynn Lyons, and Carol Taylor Fitz- 
Gibbon. Program Etfaluation Kit. Beverly 
Hills, CA: Sage Publications, 1978. 

A series of eight booklets dealing with various 
aspects of evaluating educational programs. 
The titles of these booklets are: An Evaluator*s 
Handbook, How To Deal With Goals and Ob- 
jectives, How To Design a Program Evalua- 
tion, How To Measure Program Implementa- 
tion, How To Measure Achievement, How To 
Measure Attitudes, How To Calculate 
Statistics, and How To Present an Evaluation 
Report. 

Pace, C. Robert (E >.). Evaluating Learning and 
Teaching. San Francisco: Jossey-Bass, 1973. 

A series of articles on various aspects of evalua- 
tion in higher education. The complexity of this 
issue is discussed along with descriptions of its 
more important aspects, alternative ways of 
viewing and measuring parts of it, and ^me of 
the relationships and interactions about which 
one should be aware. 

Parlett, Malcolm, and Garry Dearden (Eds.). In- 
troduction to Illuminative Evaluation: Studies 
in Higher Education. Cardiff-By-The-Sea, CA: 
Pacific Soundings Press, 1977. 

A collection of essays based on the editors* 
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work in the field of evaluation in higher educa- 
tion. An effective procedure for helping 
educators and administrators understand the 
complex processes and outconies of such a task 
is described. Data from rase studies is used 
throughout the book to support the editors* ac- 
count of what it means to be an evaluator of 
educational programs. 

Sanders, James R., and Dean Nafzigcr. A Basis 
for Determining the Adequacy of Evaluation 
Designs. Bloomington, IN: Phi Delta Kappa 
and DEDR, 1976. 

This paper suggests procedures for determining 
the adequacy of evaluation designs in advance 
of actually conducting evaluations. The authors 
have broken down the paper into four parts. In 
the first part, basic questions are considered. 
The second presents a checklist of basic con- 
siderations which are important in judging 
evaluation designs. The third presents a sample 
design. In the fourth part, noted professional 
educators present their thought about judging 
the adequacy of evaluation designs. 

Tiickman, Bruce Wayne. Evaluating Instructional 
Programs. Boston: Allyn & Bacon. 1979. 

This book gives a comprehensive treatment of 
instructional evaluation including many 
reference tools to aid administrators in carrying 
out such programs. The various phases of basic 
programs are outlined including the wnting of 
objectives, the selection, design, and judgment 



of quality of measuring instnmcnts. recording 
findings, and other aspects. Guidelines arc also 
given for formative, summative. and ex post 
facto types of evaluations. 

Walker, Jerry, Blaine Worthen, Henry Brickell, 
Michael Scriven, and Daniel Stufflebeam. 
Methodology of Evaluation. Washington, DC: 
AERA, 1974. 

A cassette recording touching on various 
aspects of evaluation methodology. Topics in- 
cluded on the cassette are: "The Influence of 
Alternative, Structural, Organizational, or 
Managerial Options on the Role of 
Evaluation**; "Content Specialization in 
Educational Evaluation: A Necessary Mar- 
riage**; "Influence of External Political Factors 
on Evaluation Role and Methods**; and two 
discussants commenting on the previous 
papers. 

Wood, Lynn, and Barbara Gross Davis. Designing 
and Evaluating Higher Education Curricula. 
Washington, DC: ERIC-AAHE, 1978. (Higher 
Education Research Report Series) 

This book provides a basis for those who are 
beginning a program or curricular evaluation m 
higher education. It has been designed for use 
by faculty or administrators to evaluate their 
current curriculum, to assess the rationale 
behind the influences that promote curriculum 
change, and to design and implement new 
courses or programs. 



This edition of RIDE was adapted 
from materials originally prepared for the 
Telemenioring Project which is housed in 
the Reich College of Education at Appala- 
chian Stale University. This FIPSE-funded 
project develops and delivers long- 
distance, state-of-the-art, continuing pro- 
fessional education and training to 
developmental educators across the na- 
tion. Representative training topics include 
content areas such as learning-and 
teaching-styles assessment and application, 
student development theory and matricu- 
lation management, teaching tips, and 
working with the adult learner. 

Proposed as a cost-effective alternative 
to traditional staff-development activities, 
this delivery system comprises the diversity 
of electronic media including computer 
conferencing, videotapes, and real-time 
satellite TV in addition to more conven- 
tional means such as printed study guides 
and conference telephones. Inquiries 
regarding participation in the project 
should be addressed to the author. 
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